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ABSTRACT 

Criterion referenced testing has received 
considerable theoretical, but only limited practical^ application* 
Grand Forks School District has developed mathematics criterion 
referenced tests for grades three to nine* The tests are keyed to a 
hierarchical sot of approximately 50 performance objectives and 4 0 
individualized contracts per grade level. These tests were 
administered on a pre-post basis during 1971-72* This study was 
designed to consider the following three concerns i (1) adoption 
experiences when using criterion referenced testing^ (2) research 
conclusions as a byproduct of this testing^ and (3) attitudes of 
classroom teachers to this^ method as compared to nationally formed 
tests* Data for the second concern indicate different orders and 
grade levels at which students learn specific skills in various 
schools* (Author) 
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AN APPLICATION OF CRJTERiON REFERENCED TESTING 



Walt Knipe, DLractor, Leafnejf Orientation to T^cbnolagy Project 
Grand Forks (North Dakota) School DiBtrlct 

Edward Krahmer, Director, Resource Information Center 
Grand For ka, North Dakota 

Introduction 

Criterion refarencad testing 1^ becoming a houiehold bywordi it is oftan refarrad 
to as the '■answer*' to education-i test and measurement problems* This is 
obviously a vary ambitious responaibllity to assign to a raiativaly recent concept. 
Behavioral objectives are another exampla of something once considered to be a 
panacea for education, which is now being seen In a more realistic light. This 
paper attampta to consider some of the more realistic possibilities for criterion 
referenced teeting in a local school setting. The results for the most part are 
preliminaf y since the project and research study from which the data contained in 
this paper were obtained is ongoing* 

Objactivea - 



Since tha intent of this paper la to present ^Gmo o£ the research byproducti of an 
application of criterion referenced testing to a mediuin sized school district, the 
obj^ctivaa are as follows: 

1. What are the adoption experiences of a school district with regard to criterion 
refereccad testing? 

2. What are the primary research conclusions resulling as byproducts from th^ 
criterion referenced experience of a school district? 

3. After two yeara experience with criterion referenced testing, what are the 
attitudes of the classroom teachers to this method o£ teiting in comparison 
to traditional national no rm^ referenced tests? 

Part 1 Adoption Experiences * 

The experience of the Grand Forks (North Dakota) School District, a dlitrlct hav^lng 
an enrollment of approximately 12, 000, with regard to criterion referenced testing 
dates back to at least i967*-68 when the first attempts were made to develop indi^ 
vlduali^ed teacher/learner contracts. The summer of .l969f the school district 
employed over a hundred teachers (using local monies) to specify in detail the per* 
formance objectives for k-*l2 in most subject areas and to begin the development of 
a comprehensive s^t of individualized teacher /learner contracts as one instructiona/ 
method whereby students sould meet thasa objectivei. This ambitious undertaking, 
now known regionally as the Grand Forks Learning System, resulted in over 3, 500 
contracts th« first year and over 4^ 000 by the present time* 

Obviously, not all grade levisls or subject areas received the same degree of 
attentior : Mathematics, however, being quite amenable to an individualized 



contracts approach, probabj.y received the most conoistant and extensive treat- 
ment by grade levels. An average oi something in excess of 50 objectives per 
grade level developed on a hierarchical skills baaii with approximately 40 con.. 
tracts per grade to rulfill these objective a resulted. 

The diatrlct adTninistration, early recognizing the nscesaity oi a management 
system for maintaining thie Learning Syitem, encouraged the mathematica depart- 
ment of one junior high 8chool to feek Title 111 ESEA support from tho State 
of North Dakota to establish a Computer atsisted and computer managed instruct- 
ional model (CAI and CMI) paralleling the Learning System, This project, Learner 
Orientation to Technoiogy (LOTT), now In its third year of operation, immediately 
focused on the area of testing as the primary preBSiag problem with regard to 
utillaing CAI, CMI, and individualized contracts. 

During the summer of 1971 a number of mathematics teachers were employed to 
develop test items to be used in nieasuring achievement of the performance objectives 
From one to three items were developed per objective; the number of items depending 
upon whether or not the objective provided a variety of options foi- Its achievement. 
Criterion r eferenced tests of approximately 120 items regulted for each grade level 
from 3 through 9. Since no overlap in objectives occurs from one grade level to 
the nejct, no overlap in test items occurred. Overlap in terms of mathematics 
skiUa, however, does occur from grade level to grade level, and, as such, is 
measured. The student ii expected to correctly answer 1 of 1. 1 of 2, or 2 of 3 
items as evidence of successful obtalnment of each objective. Each test item had 
five possible reaponses. 

Teats were pricted for each grade level of each school by the school dlitrict print 
shop, and adtnlniatored in early October of 1971, Each teacher was provided with 
an inatruction manual which explained how the tests shoiild be administered and some 
suggested uses for the .fesuits. 

Students marked th«ir responses on a mark sense computer card, which coiitained 
their name and student X, D, numbe r. 

Some of the reactions to criterion referenced testing after a two year period of time 
v/'ill be found in the conciuiion of this paper. 



ERIC 



Part 1 1 Gritof ion^Rgforohced Teat Reseafch CDScltjsLoiis 



The iQcaUy developed c^^te^ion referenced tejti have been adnilniet#f od twice 
on a district-wide basis to all studentfi in grades 3 through 8. Both adminigtra- 
tisns were during 1971-72; th^ Fall admlniitration for bateline and placefnent 
purposes and the Sprui^^administratiDn for follow-up and for placament In th# Fall 
of 1972*73, ' 4 

At the presant time the possible side benefits vvhich can result from performing 
statiatieal analysis on tiie data obtained ffom administration of criterion- 
refaranced teste are only just being reaUKed by district personnal. These side 
benafitfl are being identified as a reault of tha observation that there are obvious 
difforencea bet'ween achoola and even amoung classes within schooli as to tlie 
emphaaia placed on the order and sequenqe of etudent fulfillment of performance 
objectivea* 

While no eophlaticated statistical analysis hat yet btten performed on the datSa th 
the reiulte presented should justify tha conclusiona that^ 

a, different learning patterns eiciat fr orn ichool to school 

b. even within a achool, different learning patterne ©j^iat for dilferent clajscis. 

To demottiitrate the first conclusion, 50 fifth grade pupils ware randomly selected 
from each of three elementary achools and the same number of eighth gr aderg 
from three junior high schoalas A plctofial demonstration of the diffsrent learning 
patterns waa developed by acoring the criterion referenced tegt from the Spring 
admlniit ration and plotting correct responssi (using as asterisk) In columna 
rcpresanting performanca objectives. The foUowlng six pages indicate the reiults 
obtained* 

liooking at one example i Grade 8; it will be noted in the column headed objactlve I 
that the following data resultedj 

Numbejg of Co rrect Reaponaea percent 
34 " ' 68. 0 

23 46.0 
27 54, 0 

Looking at the list of parformance objectives measured by thii teat, it will be 
nofc^d that objectlva 1 refers othor Bagea to Baie 10^ (See Appendix A at the 
concluaion of fchia report for a list of objectives for grades 5 and 8)» Thus two 
thirds of the studemts in Scho^ol A had met this objective by the and of eighth grade^ 
while iesi than half had met it in School B, a considerable difference. 

To demonstrate the second conclusioni the sajne pictorial method was used, only 
here the students were randomly selected from the eighth grade claases of each 
teacher In one school. Tho following figuree provide thf rf^iuJtu. for two eKampla 
teachers. Looking at the last objective, Relative eri-ori students in Teacher 
A*i claaaeg met this objective in 30 cases (60%) while atudents in Teacher B^i 
classas met the objectiva 41 times (SEfe)* ' 
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Part 111 Teaeher Attitudes TG wayd Criterion P.eferanctd Tagtg 

Aiter one fuli year (1971- 72) and the fiifit semestef of 197E^73, teachers who 
had participated in the criterion referancad testing program were administarad 
in early Tabruary, a brief attitud© inventory. All elementary teachers in 
gradea 3 through 6 and junior high ferades 7-9) mathematici taacherB comprised 
tha potential raapondenti. 

The invautory contained fourtean questioni each answared by checking one of five 
choices froni StrDngly Agree to Strongly Disagree. Emphasii was placed on 
questioni which dealt with crltarion' referenced at opposed to non-refaranced 
testa* Sinca Individual item regponaai were desired, and time was e%tramely 
ihort, no effort was made to lacure reliability or validity data other than face 
validity which the res.der can judge for himeelf since the fourteen items comprising 
the inventDry appear belov/* 

1. 1 found Crltarion Reference Tests more difficult to F.dmlniater than standardized 
teeti, such as the Iowa Teits of Basic Skilli* 

2. The language level of tiie Criterion Reference Test is appropriate for the grade 
level at which I teach, 

3. The content of the Criterion Reference Test is more adaptabla to local naads, 
than Qthar standardised teats. 

4* A Grltarion Raferance Test helps me to diagnois ikill difficulties better than 
other standardized Tests, V 

5, The individual diagnoatic printout of the Criterion Raference Test ia more use- 
ful than the item analysis on standardiEed tests such as the Iowa Teats of Basic 
Skills* 

6, The group diagnostic printout of the Criterion Raferance Teat is more uiaful 
than the item analysis on other standardised tea 

7* The Qontent of tests such as the Iowa Tests of Basic Skills is more appropriate 
than the Criterion Refarence Test to test achievement of studentfl, 

8« A Criterion Reference Test in other subject areas would be useful in improving 
the educational procesi* 

9i The Criterion Reference Test can be used for pl^^ - 

10, The Criterion Refarence Test can be used at the end of the school year to 
asieii atnadent achievamantt 

U. When the Criterion Reference Test Is being used for placement purposao at the 

Test ehould be the same as the students grade level. 



• The lack of no^ms for the Criterion Referaace Test limits iti usefulness. 



13, Critarion Ref@r©nce Testa relsta cXoaely to tlie districta curriculum objectives. 

14, Critarion Reference Tests halp taachars to achieve the teaching goala of- our 
' school district. 

The following table contains the re iults obtained from atoinistering the criterion 
referenced test to the 210 potential re spondanti of which 142, or 67, 6 percent^ 
responded. The responsei ar© listed by item nwnber ralereneed to the preceeding 
Jist of items. Three columns of data are provided for each item, elementary, 
junior high, and lotaL Each line eontaliia frequency and percentaga for the responses 
aa wall as means and standard deviationa. 

As might be eKpected the elementary teachers, for whom mathematics ia only one 
of many aubjects, tend not to have reached any group decision as to the value of 
criterion relerencad tests, Appro%imateiy equal numbers reaponded Agrees 
Disagree, and Not CertaJn with the exception ol items 2 and 11 whare a significant 
number also responded Strongly Disagree and Items 9* 10', and 13 were a greater 
proportion reaponded Agree, Basically, however, the results are such that no 
meaningful conclusions can be reached, It ia suggeited that tihe elemantary teachers 
need more time and invQlvamentCiniervice trsiningp etc* ) before a more obvioua 
attitudinal pattern will become apparent. Since baseline data ia not available, no 
statiitical judgemant can be reached as to whether or not the reiulti nhown differ 
from what exiated^prior to criterion referenced teitl 

The junior high teachera, all specialii'ang in mathematics, tended to have reached 
more definits conclusioni, although even here some obvious minority opinions are 
apparent. Nearly all of the teachers agreed that appropriate language was used in 
the teats(ltem 2), Two thirds of the teachers reipended ^'favorably'* to usage of 
criterion ref eranced tests in pU^^ Tests 
of Basic Skills (items 1, 3, 4, 5, Wad 6), A similar proportion responded ''favorably" 
to usage of criterion referenced testa for placement (Item 9) and assessment (item 
10) purposes/ Finally appro3rimately^^h^ ''favorable'* 
responses (with another quarter checking Not Certain) to usage of criterion reference 
taats in oliier aubjact areas (item 8) and relationship to the district's instructional 
program (item 13 and 14), 

The remaining three item a reveal that, even among teachers who responded 
^'favorably^Vto the usage of crlterio still lingers some 

support for non- referenced teets. For instance, nearly Half of the teachera (with . 
anoiihar quarter Not Certain) felt that the content of the Iowa Teits were' more 
appropriate than the criterion-referanced tests for testing achievement (item 7) 
or tliat the lack of norms limits the usefulness of Him criterion* referenced testa 
(item 12)* 
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iData Pertaining to Eea^nBes of ElaD^Efcajy mi, Junior High School Teaeherf to the 

Detirabllity of Crltsrlon^fif wmced TestB 

ITEM 

NlDCra Group SA* A* 

1 lien 15/11.9 m/h,2 
J.H, 0/0 3/18.8 
Totsi. 15/10.6 3i/ai.8 

2 E 3/ a.t 3l/a.6 
JH 3/18,8 12/75.0 
T 6/ 43/30.3 

3 E 10/7.9 i(2/33.3 

5/31.3 5/31.3 

T 15/10,6 47/33,1 

4 £• 7/ 5.6 46/36.5 
JH ^12.5 8/50,0 
T 9/ 6.3 54/38.0 

5 B 8/ 6.3 47/37.3 
JH 3/18.8 7/43.8 
T 11/7.7 54/38.0 

6 E 2/1,6 39/31.0 
JH 2/12,5 8/50.0 
T 4/2.8 47/33.1 

7 E 5/ 4.0 36/^.6 
JH 1/ 6.3 2/12.5 
T 6/ 4*2' 38/26.8 

BE 8/ 6,3 35/27.8 

JH . 1/ 6.3 6/37.5 

T 9/ 6.-3 41/28,9 

9 E 3/2.4 55/44,0 

■ ^ ?/l3.3 7/46.7 

T 5/ 3.5 62/U.O 

10 E 4/ 3.2 54/43.2 

JH 1/ 6.7 10/66.7 

T 5/ 3.5 64/45.4 

U E 5/4.0 ^/22.4 

JH 0/0 5/33.3 

T 5/ 3.5 33/ffl.4 

12 E 9/ 7.2 35/^.0 
JH 0/0 7/46.7 
T 9/ 6.4 43/29,8 

13 E 4/ 3,2 53/42,4 
JH 4/26.7 5/33.3 
T 8/5.7 58/41.1 

14 E 4/3,2 4^33,6 
JH 2/13.3 5/33.3 
f 6/ 4.3 47/33.3 



HC* 


1/* 




A 


SD 


33/26.2 

3/i8.B 


38/30.2 


to/ q r 
1^/ 9#5 


2,97 




8/ 50.0 


Z/12«5 


All 

2,44 


C.96 


36/25.4 


46/33.4 




dmyX 


JLtJLT 






Old 

^*a8 










1 


A 






39/27.5 


Z,o5 








Xf/±3mi 




1 AA 




u/u 


3i75 


i»l3 


26/20.6 




1 '7 /1 ^ A 




i oi 
Xf^dX 








1 TO 

l»19 








3*o3 


A 

U,e9 




^ At 




4/25,0 


d/X^m3 




^ AS 


A»i3 


U/U 


3*oy ; 


A A tf 












&3/ JLy m Q 


u/u 


&#07 


Xm^l¥ 


^/ 4>»U 






A <IQ 

w*o9 






iO/-lJLiJ 


^ 0*7 


T AJ 


33/ Mmd. 




iid/ V#5 


5 




6/37,5 


1/ A ^ 




d^im 




J 1 Q 


i & /i 0 'J 
Xo/Xd* ( 




1 T A 






T A y *^ 0 


3*03 


i A^a 




1/ &#3 


^12,5 


3.19 




5d/3tmQ 


4S/x9#7 


Xd/ Omy 


3.05 


1 Al 






0 ^ A 


1- A^ 


1 7 4. *7 

1/ 6*7 


4/26*7 


1/6*7 


3o3 


x*23 






14/ 9.9 


39I2 


1*09 








^ i A 


1 i A 




Lf 6.7 


X/ O^f 


J ■ bW 


A AO 

w#99 


3i^/^fmi^ 




14/ 9#9 


0 IE 
3*15 


1#09 






•^•59 




3/20*0 
37/26.2 


4/2^-7 


^/SD^O 






34/24.1 


52/22.7 
6/ 4,8 


2.60 


1,18 


40/32.0 


33/26,4 


3.06 


1.02 


4/26.7 

44/31.2 


3/20.0 


1/6.7 


3.13 


0,99 


36/25.5 


7/5.0 


3.07 


1,01 


38/30.4 


a/i6.e 


7/ 5.6 


3.22 


0,96 


3/20.0 


3/ao.o 


0/0 


3.67 


i,n 


41/29.1 


24/17.0 


7/5.0 


3*m 


0.98 


30/24.0 


28/^.4 


11/8.8 


-3.00 


1.02 


4/26.7 


3/20.0 


1/6,7 




1.16. 


43/30,5 


31/ffi.O 


12/8.5 


3.03 


1.04 



* Data presanted as ftequan^/peresntagt 



Connlueiona 



The author.^ view eritarion relerencad tasting as a maanlngful contributioa to 
educattoii, not as a panaceii to all thm ills of educational teatirtg and measurement. 
It li tOTi6 that mora emphaeia be placad on Implementing this technique into 
practica. This paper provides aome guidslinei for so doing as 'well aa piUalls to 
avoid. In addition, Mi atudy indicate^ some of the unexpected byproducts of 
cfitarlon raf a r^need testing which in themselvea, justify the applioation of this 
toehnique. 

The foUowing are the adoption experiences of the school dlatrlct with regard to 
criterion reference testing. 

a. The language level on aome of the lower leval teats had to be. redone. 

b. The level 3 and 4 tests aho'ild be done witii large type and large 
simple diagrams. 

c* Teacharg on the average favor criterion reference tests over 
national no mied achievement tests. 

d. At first some teachers feared the teats might be used for teacher 
evaluation. 

e. Teachers favor both tte individual student printout and the group 
printout. 



f. The group printout groups students according to com 
doflelGncies, and consequently it is a good indicator for remedial 
work and placement purpoees. 

g. The tests have showed that there is a different emphasis on skill 
objectives within achoolii and grade levels in ou^ 

h. The tests provide teachers with a mathod for determining learner 
needs and accomplishments and offers a basis for measuring 
instructional accountability, which is eKtremely variable in parent- 
teacher and student confa rancas. 

: ' ■ . ' : . ■ '. ■•.;/ 

i. The fast can be given at any tima during the year for placement 
purposes, which is a problem in our dis^^ 

federally mffiUated studentB, 

our curriculum which we are abl^^ 

o ■ - . .. - - - - 

ERJC flt^^ 

. S'^^sa computar ea^dr t^ 



1. Test results tend to indieate that studants wore more eoncaraed. about 
doing v/ell on the critejplon referenc© testa than on various types of 
achievement teats* Simply, because they fait that their results on. 
the tests would not j^equlra them to repeat previously learned skllli, 

m. The "brmnchlng effect" now needs to be implemented so that battariefi 
of sub tests can be davaloped to furtih^er pin^int ttie deficlenciss 
within some of the larger skill areas. 



n. The tests should not be given in one continuous sittings about 
one hour is maximum time if you ©Kpect to get reliable teiit 
results* 

Qt All tests should be validated on a district v^ide basis with an 
outside validator in charge of the validation effort, who has 
the necessary background and ekpertise, 

p* The district is now in the process of exploring Uia poa si 
of the concept of crlterlpn reference testiag in othar subject 
ar aas such as reading and science* 

q. The criterion reference test Is the only type of test tlimt a 
school district can use to determine if it is working toward 
its curriculum goalsv Raaults to date have been encouraging 
and helpful 
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